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SUMMARY  PAGE 


THE  PROBLEM 

In  a  recent  evaluation  of  two  dichotic  listening  tasks 
(DLTs)  as  predictors  of  performance  in  Naval  Aviation 
Undergraduate  Pilot  Training,  results  based  on  one  of  five 
scoring  methods  were  presented.  The  purpose  of  this  report  is  to 
provide  DLT  performance  scores  using  all  five  scoring  methods  fur 
comparative  purposes  to  determine  the  scoring  system  most 
economical  and  efficient  for  automated  scoring  and,  most 
sensitive  to  individual  and  mean  differences. 


FINDINGS  AND  RECOMMENDATIONS 


Five  scoring  methods  are  described  which  vary  primarily  in 
thoir  treatments  of  errors,  and  in  their  consideration  of  the 
importance  of  sequence  effects.  Five  independent  analyses  of 
previously  reported  data  were  performed.  Results  were  nearly 
invariant  across  all.  five  methods;  i.e.,  intercorrelations  among 
scores  across  scoring  techniques  exceeded  0.90.  Two  of  the 
scoring  methods  are  recommended  for  purposes  of  standardizing 
future  analyses  of  DLT  performance,  one  because  of  its  simplicity 
and  ease  of  application  and  the  other,  because  it  may  have 
gr. later  sensitivity  to  differences  in  individual  performance. 
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INTRODUCTION 


Aviation  personnel  must  attend  to  various  arrays  of 
simultaneous  as  well  as  sequential  information  inputs.  The 
number  of  potential  sources  of  information  and  the  rapidity  of 
information  flow  often  require  efficient  switching  of  attention 
among  sensory  modalities  and  among  channels  within  modalities. 
Success  in  allocating  attention  to  appropriate  sources  may 
account  for  a  significant  share  of  variance  in  piloting  success. 

An  attentional  shifting  dichotic  listening  task  (DLT) 
developed  by  Gopher  (1)  and  Gopher  and  Kahneman  (2)  proved  to  be 
a  valid  predictor  of  student  performance  in  aviator  training  for 
the  Israeli  Air  Force.  The  DLT  task  requires  subjects  to 
maintain  attention  to  target  and  irrelevant  vocal  information 
presented  to  a  designated  ear,  ignore  information  presented  to 
the  other  ear,  and  subsequently,  to  recognize  or  recall  the 
target  information  presented  via  the  designated  ear.  The 
reported  merits  of  the  DLT  have  gained  the  regard  of  a  number  of 
investigators  (4,  5,  6),  many  of  whom  are  attempting  to  increase 
the  efficiency  of  test  batteries  used  to  predict  success  for 
students  entering  military  flight  training  programs  in  the  United 
States . 

In  a  recently  reported  U.S.Navy  study  (3),  results  indicated 
that  a  dichotic  listening  task  similar  to  the  one  used  by  Gopher 
(1)  accounted  for  a  statistically  significant  portion  of  pass- 
fail  variance  in  the  Navy  flight  training  program.  The  results 
were  based,  however,  on  only  one  of  five  scoring  methods 
developed  by  the  investigators  at  the  Naval  Aerospace  Medical 
Research  Laboratory  (NAMRL).  The  primary  variations  among  these 
scoring  techniques  rels<te  to  (a)  the  treatment  of  error  scores 
(intrusions,  omissions,  etc.)  and  (b)  the  treatment  of  sequence 
effects.  Table  1  provides  a  description  of  the  contrasting 
features  of  the  five  scoring  schemes.  It  is  conceivable  that  the 
application  of  different  scoring  methods  might  yield  different 
conclusions.  The  purpose  of  this  study  is  to  examine  this 
possibility  and  document  the  correlational  relationships  between 
the  various  scoring  methods. 
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TABLE  1 


DLT  Scoring  Methods  and  Performance  Measures 


1 


Scoring  Method  Number 


Performance  Measures 


1.  An  error-based,  sequence-dependent 
scoring  method  with  differentiation 
error  type.  These  errors  are  labeled 
ommissions,  intrusions,  and  other 
(after  Gopher  &  Kahneman  (2)).  An 
omission  error  is  the  failure  to 
report  a  relevant  digit  in  a 
designated  ear.  An  intrusion  error 
is  the  reporting  of  a  digit  from  the 
nondes ignated  ear.  "Other”  errors 
include  the  reporting  of  a  letter  or 
a  digit  when  one  is  not  presented  or 
incorrectly  reporting  a  digit.  All 
responses  are  scored. 


2.  A  modified  error-based  sequence- 
dependent  scoring  method  highly 
similar  to  scoring  method  1  with  only 
a  slight  variation  in  the  treatment 
of  intrusion  errors.  In  this  scoring 
method  an  intrusion  error  normally 
results  .in  one  additional  (omission) 
error . 

3.  A  modified  sequence- i ndependent 
simple  number  correct  scoring  method. 
All  responses  are  scored  regardless 
of  sequence  or  the  number  of  respon¬ 
ses. 

4.  A  modified  sequence- i ndependent 
simple  number  correct  scoring  method 
where  only  the  first  five  responses 
of  Part  1  and  the  first  four  respon¬ 
ses  of  Part  2  of  each  DLT  trial  are 
considered . 


*  Part  1  omissions, 
intrusions  and  other 
error 

*  Part  2  omissions, 
intrusions  and  other 
errors 

*  Part  1  number  correct  = 
180  -  Total  Part  1 
errors 

*  Part  2  number  correct  = 
144  -  Total  Part  2 
errors 

*  Total  correct  =  Part  1 
number  correct  Part  2 
number  correct  (nine 
performance  measures) 

*  A  total  of  9 
performance  measures 
similar  to  scoring 
method  1 


*  Part  1  number  correct 
Part  2  number  correct 
Total  number  correct 


*  Three  performance 
measures  similar  to 
scoring  method  3 


5.  A  strict  sequence-dependent  simple 
number  correct  scoring  method.  Once 
an  error  occurs  no  additional  cor¬ 
rect  responses  are  awarded.  Part  1 
and  Part  2  responses  are  scored 
independently. 

Examples  of  the  results  of  the  application  of  each  of  the 
different  scoring  systems  are  presented  at  Appendix  A. 


*  Three  performance 
measures  similar  to 
scoring  method  3 
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METHOD 


Apparatus  and  Procedure.  The  DLTs ,  extensively  described 
elsewhere  (3),  consisted  of  the  dichotic  presentation  of  letter¬ 
digit  strings.  The  subject  was  instructed  to  maintain  attention 
to  one  ear  while  ignoring  the  information  presented  to  the  other 
ear,  and  subsequently,  to  record  on  an  answer  sheet  the  digits 
presented  to  the  designated  ear,  accurately  and  in  the  sequence 
of  occurrence.  The  test  apparatus  comprised  a  dual-channel  tape 
recorder,  headphones,  and  a  paper-and-pencil  answer  booklet.  A 
VOTRAX  synthetic  speech  system  was  utilized  to  generate  the 
auditory  stimuli  at  sound  pressure  levels  of  approximately  75 
dB/Leq  (re.:  20  n  P) . 

Each  DLT  trial  was  divided  into  two  parts  (see  Figure  1). 

The  Part  1  task  consisted  of  a  mix  of  letters  and  digits 
delivered  to  each  ear.  Digits  were  never  presented 
simultaneously  to  the  two  ears,  and  no  digit  was  repeated  in 
either  sequence.  However,  simultaneous  presentations  were 
presented  of  identical  or  dissimilar  letters,  or  a  letter  to  one 
ear  and  a  digit  to  the  opposite  ear.  Part  2  of  each  trial 
consisted  of  the  simultaneous  presentation  of  two  letters  to  each 
ear  followed  by  a  string  of  four  successive  digits.  Part  1  and 
Part  2  of  each  trial  were  each  preceeded  by  a  ’"right"  or  "left" 
vocal  command,  signifying  the  designated  ear.  The  auditory 
stimuli  were  presented  at  the  rate  of  one  letter  or  digit  per  0.9 
second.  A  single  trial,  including  pause  time,  lasted 
approximately  27  seconds.  One  hundred  eighty  total  correct 
responses  were  possible  for  Part  1,  144  for  Part  2,  over  36 
trials.  A  diagram  of  a  sample  DLT  trial  is  at  Appendix  B. 


PART  1 

Left  Ear  R  8  NSMY2GB7FL6RL5 

"Right"  (Vocal  Channel  "attend"  Command) 

Right  Ear  YL3SR4F  Z9XF0FN1L 


PART  2 

Left  Ear  B  F  4  3  7  9 

"Left"  (Vocal  Channel  "attend"  Command) 
Right  Ear  G  L  1  5  6  2 


Figure  1.  DLT  Trial  Example 


Methodological  departures  from  the  original  Israeli  DLT  were 
(a)  the  use  of  letter  text  rather  than  Hebrew  verbs,  (b)  the  use 
of  "left"  and  "right"  vocal  channel  "attend"  commands  presented 
stereophonically  (rather  than  tones  presented  monaurally  to  the 
"attend"  ear),  and  (c)  the  use  of  computer  generated  speech  with 
simultaneous  stimulus  onset  times,  left  and  right  ear,  (rather 
than  the  tape  recorded  voice  of  a  female  speaker),  and  in 
addition,  (d)  the  requirement  for  written,  rather  than  oral, 
responses . 
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Preliminary  research  suggested  that  the  DLT  lacked 
sufficient  difficulty.  A  number  of  attempts  to  increase  the 
level  of  difficulty  were  tried  (e.g.r  tones  as  channel  attend 
commands/  background  "party"  distracting  speech,  and  varying 
loudness  levels),  though  unsuccessfully.  Finally,  it  was 
demonstrated  that  incorporating  irrelevant  background  materials 
(digits  recorded  in  reverse — "zero"  becomes  "orez")  to  each 
channel  at  a  sound  pressure  level  equal  to  that  of  the  relevant 
test  material  significantly  increased  test  error.  The  initial 
DLT  and  the  DLT  containing  background  material  were  designated 
the  Clear  DLT  (CDLT)  and  the  Background  DLT  (BDLT) ,  respectively. 
The  only  difference  between  the  two  was  that  the  added 
background  material  was  applied  to  the  BDLT. 

Subjects.  Ninety-four  male  Navy  and  Marine  Corps  student 
naval  aviators  (SNAs)  awaiting  assignment  to  undergraduate 
training  volunteered  to  participate  in  the  study.  Seventy 
subjects  were  Marine  SNAs  from  the  Marine  Aviation  Training 
Support  Group  at  the  Naval  Air  Station,  Pensacola,  Florida  and  24 
were  Navy  SNAs  from  the  Naval  Aviation  Schools  Command  at  the 
Naval  Air  Station,  Pensacola,  Florida. 

The  CDLT  sample  consisted  of  12  Navy  and  34  Marine  Corps 
SNAs.  The  BDLT  comprised  12  Navy  and  36  Marine  Corps  SNAs. 

RESULTS 

Each  of  the  five  scoring  methods  described  in  Table  1  was 
applied  to  the  DLT  scores  for  each  subject.  The  correlations 
among  the  various  performance  parameters  (number  correct, 
omission  errors,  intrusion  errors,  etc.)  based  on  scoring  method 
1  for  CDLT  and  BDLT  are  presented  in  Tables  2  and  3, 
respectively.  Most  of  the  intercorrelations  are  quite  strong 
with  the  exception  of  Table  3  "other  errors".  "Other  errors," 
incidently,  comprise  the  smallest  percentage  of  the  various 
error  categories  for  both  the  CDLT  and  the  BDLT,  based  on 
scoring  system  1.  "Omissions"  are  the  largest  error  category 
(CDLT  51%,  BDLT  58%)  followed  by  "Intrusions"  (CDLT  33%,  BDLT 
29%)  and  then  "Other  errors"  (CDLT  16%,  BDLT  12%).  The  magnitude 
of  the  correlations  suggest  that  except  for  "other  errors"  of  the 
BDLT  the  various  performance  scores  are  in  general  agreement. 
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Table  2 

CDLT  Intercorrelation  Matrix  (Scoring  Method  1) 

DLT  Measures  1234  56789 


Part  1 

Correct 

1 

- 

Part  2 

Correct 

2 

.922 

- 

Total  Correct 

3 

.982 

.978 

Part  1 

Omission 

4 

-.975 

-  .926 

Error 

Part  1 

Intrusion 

5 

-.932 

-.824 

Error 

Part  1 

Other 

6 

-.687 

-.613 

Error 

Part  2 

Omission 

7 

-.887 

-.981 

Error 

Part  2 

Intrusion 

8 

-.857 

-.966 

Error 
Part  2 

s 

Other 

9 

-.743 

-.688 

Errors 


973 

- 

899 

-.863 

- 

665 

.605 

.550 

- 

950 

.906 

.787 

.534 

- 

927 

.803 

.762 

.574 

.951 

.731 

.702 

.690 

.632 

.571  .530 

Table  3 

BDLT  Intercorrelation  Matrix  (Scoring  Method  1) 

DLT  Measures  1234  56789 


Part  1  Correct 

1 

- 

Part  2  Correct 

2 

.710 

- 

Total  Correct 

3 

.914 

.935 

Part  1  Omission 

4 

-.946 

-.776  -.925 

- 

Error 

Part  1  Intrusion 

5 

-.883 

-.581  -.780 

.747 

Error 

Part  1  Other 

6 

-.340 

-.044  -.146 

.086 

.287 

Error 

Part  2  Omission 

7 

-.713 

-.965  -.922 

.791 

.549 

.  040 

— 

Error 

Part  2  Intrusion 

8 

-.626 

-.953  -.866 

.696 

.538 

.136 

.881 

Errors 

Part  2  Other 

9 

-.263 

-.205  -.251 

.170 

.301 

.405 

-26  .075 

Errors 


The  primar”  statistical  concern  focuses  upon  the  family  of 
relationships  among  scores  derived  under  the  five  differing 
scoring  techniques.  Because  of  the  large  number  of  performance 
measures  associated  with  the  scoring  methods  and,  as  reported 
above,  because  of  the  strong  relationships  among  performance 
data,  the  scores  derived  under  each  scoring  method  were  collapsed 
to  provide  a  single  total  number  correct  score  for  purposes  of 
analysis.  Normative  values  for  the  scoring  methods  are  presented 
at  Table  4.  Repeated  measures  analysis  of  variance  statistical 
treatments  indicate  significant  differences  between  scoring 
methods.  For  the  CDLT  [F(4,180)  =  33.94;  £  <  .01]  ,  scoring 
method  5  resulted  in  significantly  more  errors,  while  scoring 
methods  one,  two,  three,  and  four  produced  similar  results.  For 
the  BDLT  [F  (4,188)  =  178.55;  £  <  .01]  ,  methods  two  and  four, 

and  three  and  four  produced  similar  results.  All  other  scoring 
method  comparisons  were  significantly  different. 

The  ratio  of  standard  deviation  to  mean  provides  one  measure 
of  the  sensitivity  of  the  various  scoring  techniques  to 
individual  differences  in  performance.  Using  this  measure  as  a 
standard  (see  Table  4)  scoring  method  5  is  more  sensitive  to 
individual  variation  for  both  the  CDLT  and  BDLT. 

More  important  than  comparison  of  normative  values,  the  high 
intercorrelations  (all  are  0.9+)  among  total  correct  scores  (see 
Table  5)  indicate  that  the  rank  ordering  of  scores  is  generally 
insensitive  to  the  scoring  technique  employed.  Undoubtedly,  the 
high  degree  of  similarity  between  scoring  methods  one  and  two  and 
methods  three  and  four  contributed  to  the  high  correlation  values 
for  these  measures.  The  high  positive  correlations  indicate,  in 
other  words,  that  a  hypothetical  rank  ordering  of  the  performance 
scores  for  subjects  ranging  from  lowest  to  highest,  remains 
intact  regardless  of  the  scoring  technique  selected. 
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Table  4 


Means,  Standard  Deviations  and  Standard  Deviation/Mean 
Ratios  of  DLT  Performance  Scores  Obtained 
with  the  Five  Scoring  Methods 


Mean 

Standard  Deviation 

St .Dev./ 
Mean 

Clear  DLT 
(46  subjects) 
Scoring  Method 

1 

315.0 

14.8 

.047 

2 

316.2 

11.8 

.037 

3 

318.8 

9.5 

.030 

4 

317.9 

9.7 

.031 

5 

305.3 

24.0 

.079 

Background  DLT 
(48  subjects) 
Scoring  Method 


1 

277.9 

28.3 

.102 

2 

286.8 

21.0 

.073 

3 

294.7 

18.3 

.062 

4 

291.7 

19.0 

.065 

5 

250.1 

36.1 

.144 

Table  5 

Xntercorrelat 

ions  Among  DLT 

Performance  Scores 

Obtained 

with  the  Five 

Scoring 

Methods 

Scoring 

Method  1 

2 

3 

4 

5 

Clear  DLT 

(45  subjects)  1 

2 

.995 

- 

3 

.983 

.974 

- 

4 

.994 

.991 

.990 

- 

5 

.975 

.983 

.952 

.978 

- 

Background  DLT 

(48  Subjects)  1 

2 

.984 

- 

3 

.974 

.974 

- 

4 

.977 

.987 

.980 

- 

5 

.904 

.945 

.900 

.946 

- 

;v 

.v 

in 

* 
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CONCLUSIONS 


Based  on  the  high  positive  correlations  between  the  various 
scoring  methods,  it  must  be  concluded  that  individual  variation 
is  generally  insensitive  to  the  scoring  method  employed.  Thus, 
the  selection  of  scoring  technique  probably  has  no  bearing  on  the 
interpretations  or  generalizations  derived  from  application  of 
the  selected  technique.  This  means  that  the  conclusions  provided 
by  Griffin  and  Mosko  (3)  are  confirmed. 

The  simplicity  of  scoring  method  3  (automated  or  manual), 
however,  makes  it  the  more  attractive  technique  from  the 
standpoint  of  economy.  This  method  is  a  sequence- independent 
response  technique.  Although  the  technique  does  not 
differentiate  error  type,  it  is  completely  objective,  and  it  is 
relatively  easy  to  score  as  Table  1  suggests.  Scoring  method  5, 
on  the  other  hand,  is  appealing  because  of  its  apparently  higher 
sensitivity  to  individual  variation  as  measured  by  the  ratio  of 
its  standard  deviation  to  mean.  This  scoring  method,  unlike 
method  3,  is  seguence-dependent  (i.e.,  only  correct  responses 
preceding  initial  errors  in  Part  1  and  Part  2  are  awarded). 
Scoring  method  5  is  completely  objective  although  decidedly  more 
difficult  to  score  than  is  scoring  method  3. 
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APPENDIX  A 

Results  of  the  Application  of  Table  I  DLT  Scoring  Systems 


P  A  R  T  1 


P  A  R  T  2 


Scoring  System  I  Channel 

Error-based,  sequence  Command 
dependent  with  error  "LEFT" 

differentation  i.e. 

(omission,  intrusion, 
and  other  errors) 


ttend'  Left  ear  8  2  7  6  5  r  • 

LEFT*  Response  [7]  VT\  \T\  tTl  \T]  1.6  "RIGHV0]  \J]  [T|  |T]  2 
Right  ear  stW  V  1  5  6  2^ 

'  _  l.'  _  inti*! 


Scoring  Result 

3  errors  Part  1 , 
1  error  Part  2 


omission 

error 


other  intrusion 
error  error 


intrusion 

error 


Loft  ear  8 


Scoring  System  2  Laft  ear 
Simi liar  to  System  1  „.  FFr..  opsnonse 
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